Web Survey Bibliography
Title Do not touch data from late-received online questionnaires: Analysis of late responder in a closed-pool online survey
Author Stieger, S., Voracek, M.
Year 2005
Access date 22.04.2005
Abstract In postal surveys paper-pencil questionnaires are provided only for a certain time-frame. Normally, data collection lasts for some days, weeks or months. However, it is not clear when to stop data collection and there are no guidelines. This is also of relevance from a methodological point of view. If the expected effect-size is low, large number of participants have to be asked in order to get statistically significant results. To get a large number of participants, either a large number of potential participants has to be recruited or the study has to be online for a certain time. The longer a study is online, the more likely statistical significance will be reached. However, over time the possibility of a negative influence through late-responder increases. In our study, we therefore investigated whether late-responder differ from early-responder. Participants in this online study were recruited via email (addresses were retrieved from a publicly accessible directory). After sending out the invitation for participation via email, the return rate decreased very quickly, but did never reach the zero-level. Even after one year, online questionnaires from participants were still received (late-responder). Based on 4554 online questionnaires, we show that, on average, after the 45th day of the online study (1) the number of participants which masked their sex (gender-switching) increased strongly, (2) the relative distribution of participants' faculty affiliation changed notably, (3) participants more often tried to ignore required fields, (4) the portion of participants clicking through the online survey stayed constant, but the portion of questions clicked through increased, (5) the questionnaire completion time decreased and, at the same time, less questions were answered, and (6) the page-specific dropout rate increased.
Abstract - optional Üblicherweise werden Paper-Pencil-Fragebögen in postalischen Umfragen in einem zeitlich begrenzten Rahmen vorgegeben, d.h., die Erhebungsphase beträgt einige Tage, Wochen oder Monate. Generell ist jedoch nicht klar, wann eine solche Erhebung abgebrochen werden soll bzw. gibt es dafür keine Richtwerte. Diese Frage ist nicht zuletzt auch methodisch von Bedeutung. Um auch bei kleinen erwarteten Effektgrößen zu statistisch signifikanten Ergebnissen zu kommen, ist es notwendig, eine große Anzahl an Versuchspersonen zu befragen. Um zu einer großen Anzahl an Versuchspersonen zu gelangen, muss entweder eine große Anzahl an Personen rekrutiert werden oder die Studie entsprechend lange laufen. Je länger eine Studie läuft, umso eher wird statistische Signifikanz erreicht, jedoch auch umso wahrscheinlicher ist eine mögliche negative Beeinflussung der Datenqualität durch Late Responder. In dieser Online-Studie untersuchten wir deshalb, ob Versuchspersonen, welche spät auf eine Teilnahmeaufforderung reagieren, sich von jenen unterscheiden, welche zu Beginn der Studie teilgenommen haben. Die Versuchspersonen in dieser Online-Studie wurden via Email rekrutiert (die Adressen stammten aus einem öffentlichen Adressbuch). Nach der Aussendung zeigte sich, dass zwar der Rücklauf sehr schnell absank, jedoch nie ganz auf Null zurückging. Selbst nach mehr als einem Jahr nach Untersuchungsbeginn trafen noch immer Fragebögen von Versuchspersonen ein, die eine Teilnahmeaufforderung erhalten hatten (Late-Responder). Anhand einer Analyse von 4554 Fragebögen kann gezeigt werden, dass im Durchschnitt nach dem 45. Tag des Beginns der Online-Studie (1) die Anzahl der Versuchspersonen, welche ihr Geschlecht maskierten (Gender Switching), stark anstieg, (2) die relative Fakultätsverteilung der Versuchsteilnehmer sich stark veränderte, (3) die Versuchspersonen vermehrt versuchten, Required-Fragen zu umgehen, (4) der Anteil an Durchklicker zwar gleich blieb, jedoch die Anzahl durchgeklickter Fragen stark stieg, (5) die Ausfüllzeit des Fragebogens sank, bei gleichzeitig weniger ausgefüllten Fragen und (6) der seitenspezifische Dropout stark zunahm.
Access/Direct link Homepage - conference (abstract)
Year of publication2005
Bibliographic typeConferences, workshops, tutorials, presentations
Web survey bibliography - 2005 (76)
- The ethics of research using electronic mail discussion groups; 2005; Kralik, D., Warren, J., Koch, T., Pignone, G., Price, K.
- The Analyses of Domestic Study about Internet Survey; 2005; Rui, L., Tie-ying, S.
- Controlling the Baseline Speed of Respondents: An Empirical Evaluation of Data Treatment Methods of...; 2005; Mayerl, J.
- Determinanten der Rücklaufquote in Online-Panels; 2005; Batanic, B., Moser, K.
- On the cost-efficiency of probability sampling based mail surveys with a Web response option; 2005; Werner, P.
- Expert workshop on mixed mode data collection in comparative social surveys; 2005; Roberts, C.
- The Effect Of A Simultaneous Mixed-Mode (Mail And Web) Survey On Respondent Characteristics And Survey...; 2005; Brennan, M.
- The total survey error approach. A guide to the new science of survey research; 2005; Weisberg, H. F.
- The professional respondent problem in online panel surveys today; 2005; Fulgoni, G.
- Satisficing behavior in online panelists; 2005; Downes-Le Guin, T.
- Reading behavior in the digital environment: Changes in reading behavior over the past ten years; 2005; Liu, Z.
- Rating versus comparative trade-off measures. Trending changes in political issues across time and predictive...; 2005; Thomas, R. K., Behnke, S., Johnson, Al., Sanders, M.
- Publication bias: Recognizing the problem, understanding its origins and scope, and preventing harm; 2005; Dickersin, K.
- Panel proliferation and quality concerns; 2005; Faasse, J.
- Gricean effects in self-administered survey. Ph.D. Dissertation; 2005; Yan, T.
- Drop-down boxes, radio buttons, or fill-in-the-blank? Web survey scale-type effects; 2005
- Does weighting for nonresponse increase the variance of survey means?; 2005; Little, R. J., Vartivarian, S.
- Big scale observations gathered with the help of client side paradata; 2005; Haraldsen, G., Kleven, O., Sundvoll, A.
- User Interface Design and Evaluation ; 2005; Stone, D., Jarrett, C., Woodroffe, M., Minocha, S.
- Adding Value to Data Through Improved Access. The Case for Web Portals; 2005; Baker, R. P.
- Multi-Mode Research and Data Linkage. Theoretical and Practical Advice; 2005; Terhanian, G.
- Architectural Design of a Survey Questionnaire and Respondent Data Repository. Practical Considerations...; 2005; Cookson, P., Sobell, J.
- Developing and validating a nursing website evaluation questionnaire; 2005; Tsai, S. - L., Chai, S.-K.
- Workaround: Site’s surveys beat pop-up blockers, yield responses; 2005; Arnold, C.
- The Story of Subject Naught: A Cautionary but Optimistic Tale of Internet Survey Research; 2005; Konstan, J. A., Ross, M. W., Rosser, B. R. S., Stanton, J. M., Edwards, W. M.
- Standards in Online Surveys. Sources for Professional Codes of Conduct, Ethical Guidelines and Quality...; 2005; Kaczmirek, L., Schulze, N.
- Computer adaptive testing; 2005; Gershon, R. C.
- Ego control and ego-resiliency: Generalization of self-report scales based on personality descriptions...; 2005; Block, J., Funder, D. C., Letzring, T. D.
- The Web experiment list: A Web service for the recruitment of participants and archiving of Internet...; 2005; Reips, U. -D., Lengler, R.
- Survey of substance use among high school students in Taipei: Web-based questionnaire versus paper-and...; 2005; Wang, Y. C., Lee, C. M., Lew-Ting, C. Y., Hsiao, C. K., Chen, W. J.
- Web Surveys. A Brief Guide on Usability and Implementation Issues; 2005; Kaczmirek, L.
- An assessment of measurement invariance between online and mail surveys ; 2005; Deutskens, E., de Ruyter, K., Wetzels, M.
- E-mail versus Web survey response rates among health education professionals; 2005; Kittleson, M. J., Brown, S. L.
- Toward An Open-Source Methodology: What We Can Learn From The Blogosphere; 2005; M.
- Aux Abonnes Absents: Liste Rouge Et Telephone Portable Dans Les Enquetes En Population Generale Sur...; 2005; Beck, F., ., Peretti-Watel, P.
- Web Versus Paper Questionnares: A Design and Functionality - Comparison; 2005; Jones, Ja., Fraser, C., Dowling, Z.
- Web Surveys and the new Disability Discrimination Act; 2005; Macer, T.
- Mixed-mode Surveys Using Mail and Web Questionnaires; 2005; Meckel, M., Baugh, P., Walters, D.
- Sampling procedure, questionnaire design, online implementation; 2005; Jackob, N., Arens, J., Zerback, T., Jowell, R., de Rouvray, C.
- Simple Approaches to Estimating the Variance of the Propensity Score Weighted Estimator Applied on Volunteer...; 2005; Isaksson, A., Lee, S., de Rouvray, C.
- Simple Approaches to Estimating the Variance of the Propensity Score Weighted Estimator Applied on Volunteer...; 2005; Isaksson, A., Lee, S.
- Alternative Modes for Health Surveillance Surveys: An Experiment with Web, Mail, and Telephone; 2005; Link, M. W., Mokdad, A.
- An Experimental Comparison Of Web And Telephone Surveys; 2005; Fricker, S., Galesic, M., Tourangeau, R., Yan, T.
- Organizational Virtual Communities: Exploring Motivations Behind Online Panel Participation; 2005; Daugherty, T., Lee, W.-N., Gangadharbatla, H., Kim, K., Outhavong, S.
- Promoting Uniform Question Understanding in Today's and Tomorrow's Surveys; 2005; Conrad, F. G., Schober, M. F.
- Is a Web survey as effective as a mail survey? A field experiment among computer users; 2005; Kiernan, N. E., Kiernan, M., Oyler, M. A., Gilles, C.
- The effect of personalization on response rates and data quality in web surveys; 2005; Heerwegh, D., Vanhove, T., Matthijs, K., Loosveldt, G.
- When Methodology Interferes With Substance; 2005; Schoen, H., Faas, T.
- Web-based and Mailed Questionnaires: A Comparison of Response Rates and Compliance; 2005; Baelter, K., Balter, O., Fondell, E., Trolle-Lagerros, Y.
- Bleeding Edge or Proven Technology? The Fact and the Fiction of Mobile Survey Computing; 2005; Cameron, M. R.